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Abstract 


There  is  an  enormous  range  of  computational  models  and  simulations  for  addressing  a  variety  of 
analysis  and  design  issues  in  complex  systems.  This  report  addresses  systems  where  behavioral  and 
social  phenomena  are  significant  elements  of  system  performance.  The  report  begins  with  an  overview 
of  the  state  of  the  art  for  multi-level  modeling  of  such  "socio-technical"  systems.  It  then  reports  on  an 
interview  study  of  how  four  non-defense  industries  address  computational  modeling  of  complex 
aerospace,  automotive,  building  equipment,  and  semi-conductor  systems.  The  report  concludes  with 
several  observations  on  the  overall  study  of  which  this  work  was  an  element. 
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Introduction 


The  Department  of  Defense  has  made  enormous  investments  over  several  decades  in  developing 
computational  models  and  simulations  for  complex  military  systems  ranging  from  weapon  platforms  to 
operational  military  organizations.  The  result  has  been  an  estimated  8,000  software  artifacts,  totaling 
approximately  10,000,000  lines  of  code,  for  roughly  $10  billion  dollars  in  investment.  Not  surprisingly, 
DoD  would  like  to  reuse  these  assets  to  address  new  questions.  Consequently,  the  Assistant  Secretary  of 
Defense  (R&E)  requested  the  effort  summarized  by  the  following  Terms  of  Reference  for  the  Dynamic 
Multi-Leveling  Modeling  Framework  (DMMF). 


Summary  of  Terms  of  Reference 

"/  request  the  Modeling  and  Simulation  Coordination  Office  (MSCO)  lead  a  study  to  investigate  how  a 
single  or  limited  number  of  modeling  and  simulation  (M&S)  framework(s)  could  be  developed  to  allow 
models  from  the  engineering  to  theater  level  to  more  easily  interoperate.  The  purpose  of  the  study  is  to 
determine  if  it  is  possible  to  develop  a  tool-set  to  enable  senior  DoD  leaders  to  evaluate  the  effects  of 
new  or  modified  military  systems,  capabilities,  force  structure,  or  tactics  across  a  range  of  scenarios, 
systems,  and  fidelities.  The  goal  is  an  analysis  capability  with  the  following  characteristics: 

□  A  single  framework  (or  small  number  of  frameworks)  that  allows  interoperability  of  various  tool 
sets 

□  Operates  at  all  levels  of  analysis  from  Engineering  to  Theater 

Simultaneous  operation  of  deterministic  and  stochastic  models 
-  Incorporates  the  range  of  models  from  physics-based  to  behavioral  and  social 

□  Allows  composability,  so  the  analysis  can  be  guickly  reconfigured  to  address  a  range  of  options. 

□  Permitting  users  to  modify  parameters  at  any  combination  of  levels  and  examine  systems  / 
capability  trades 

□  Is  operable  from  desktop  or  single  location" 


Summary  of  Report 

This  report  begins  with  an  overview  of  the  state  of  the  art  for  multi-level  modeling  of  socio-technical 
systems.  It  then  reports  on  an  interview  study  of  how  four  non-defense  industries  address 
computational  modeling  of  complex  aerospace,  automotive,  building  equipment,  and  semiconductor 
systems.  The  report  concludes  with  several  observations  of  on  the  overall  DMMF  study  of  which  this 
work  was  an  element. 
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Multi-Level  Modeling  of  Socio-Technical  Systems 


Socio-technical  systems  involve  behavioral  and  social  aspects  of  people  and  society  that  interact  with 
technical  aspects  of  organizational  structure  and  processes  --  both  engineered  and  natural  —  to  create 
organizational  outcomes  and  overall  system  performance.  These  types  of  systems  are  often  also 
characterized  as  complex  adaptive  systems  where  independent  agents  pursue  their  individual  objectives 
while  learning  and  adapting  to  evolving  system  structures  and  behaviors. 

Design  and  evaluation  of  such  systems  can  be  addressed  using  the  multi-level  modeling  framework 
shown  in  Figure  1.  This  framework  explicitly  represents  the  different  levels  of  abstraction  underlying 
system  behaviors  and  performance.  People  can  only  execute  work  practices  that  are  supported  by 
delivery  operations,  which  only  exist  if  the  organizations  within  the  system  structure  invest  in  and 
sustain  these  capacities,  which  they  will  only  do  if  the  domain  ecosystem  incentivizes  and  rewards  the 
outcomes  of  these  investments. 


Figure  1.  Multi-Level  Modeling  Framework 

The  domain  ecosystem  -  society  -  defines  the  objectives  for  the  system  and  the  rules  of  the  game.  This 
includes  explicit  or  implicit  specification  of  what  matters,  what  can  and  cannot  be  done,  and  how 
performance  is  rewarded.  These  specifications  incentivize  or  impede  organizational  decisions. 

These  decisions  include  the  nature  of  system  capacities  considered,  levels  of  investments  in  these 
capacities,  and  assessments  of  subsequent  performance.  In  this  way,  delivery  operations  are  created 
and  sustained.  They  also  may  be  impeded  as,  for  example,  government  price  controls  can  lead  to 
disinvestment  in  capacities. 

Delivery  operations  provide  capacities  for  work.  These  capacities  can  include  engineered  systems  (e.g., 
networks  and  databases,  devices  and  platforms),  processes  (e.g.,  procedures,  plans),  and  venues  (e.g., 
factories,  playing  fields).  Work  practices  or  activities,  at  the  bottom  of  Figure  1,  can  include  physical 
manipulation  (e.g.,  lifting,  carrying,  controlling),  information  provision  (e.g.,  informing,  advising)  or  social 
interaction  (e.g.,  talking,  performing). 
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The  four  levels  in  Figure  1  represent  different  levels  of  abstraction.  Within  each  level,  there  can  also  be 
levels  of  aggregation,  as  illustrated  by  Figure  2.  For  example,  individuals,  teams,  specialties  (e.g., 
electricians)  or  whole  workforces  can  perform  work.  Processes  can  be  specific  sets  of  steps,  generic 
sequences  of  functions,  or  composite  procedures  for  all  automobiles  or  patients.  Organizations  can  be 
departments,  divisions,  subsidiaries  or  whole  corporations.  The  "grain  sizes"  of  the  networks  at  each 
level  reflect  the  level  of  aggregation  of  the  representation  of  the  phenomena  at  that  level. 

Networks  of  Phenomena 


Intro-Level  Information  Flow  &  Incentives 


Figure  2.  Networks  of  Phenomena  at  Each  Level 

There  can  be  a  range  of  socio-technical  phenomena  represented  in  Figures  1  and  2.  At  the  people  level, 
the  phenomena  of  interest  are  usually  human  behavior  and  performance  —  individually,  in  teams,  or  in 
groups.  For  the  process  level,  the  central  socio-technical  phenomenon  is  the  social  networks  that 
enable  processes.  The  organization  level  is  typically  concerned  with  economic  decision  making,  drawing 
upon  classical  microeconomics  or,  more  recently,  behavioral  economics.  A  good  example  of  socio- 
technical  phenomena  at  the  ecosystem  level  is  the  evolution  of  social  and  cultural  norms.  There  is  a  rich 
set  of  mathematical  and  computational  models  that  can  be  drawn  upon  to  represent  the  range  of 
phenomena  outlined  here. 

Table  1  shows  how  the  multi-level  modeling  framework  can  be  applied  in  three  different  domains. 
Application  of  the  framework  involves  representing  the  phenomena  at  each  level,  choosing  models  to 
represent  these  phenomena,  selecting  computational  means  to  operationalize  these  models  across 
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levels,  including  the  flow  of  information,  e.g.,  on  incentives,  within  and  between  levels  as  indicated  in 
Figure  2.  All  of  these  components  provide  the  "engine"  for  developing  interactive  visualizations  to 
enable  exploration  of  alternative  system  designs  at  multiple  levels,  e.g.,  process  designs  vs.  policy  rules. 


Level 

Healthcare  Delivery 

Energy  Consumption 

Military  Operations 

Domain 

Ecosystem 

Social  Priorities, 
Medicare/Medicaid 

Public  Service 

Commission 

Military  Priorities,  Rules  of 
Engagement 

System  Structure 

Providers,  Payers, 
Suppliers 

Utilities,  Builders, 
Contractors 

Commanders,  Service 
Components 

Delivery 

Operations 

Care  Capabilities,  Health 
Information 

Generation,  Trans,  & 
Distribution 

Strategies,  Tactics,  Battle 
Plan 

Work  Practices 

Patient-Clinician 

Interactions 

End-User  Consumption 

Movement  of  Forces, 
Platforms,  Etc. 

Table  1.  Comparison  of  Domains 

The  resulting  interactive,  computational  model  can  be  termed  a  "policy  flight  simulator."  Such 
simulators  can  provide  the  means  to  explore  a  wide  range  of  possibilities,  thereby  enabling  the  early 
discarding  of  bad  ideas  and  refinement  of  good  ones.  This  enables  "driving  the  future  before  writing  the 
check."  One  would  never  develop  and  deploy  an  airplane  without  first  simulating  its  behavior  and 
performance.  However,  this  happens  all  too  often  in  organizational  decision  making  in  terms  of  policies, 
strategies,  plans,  and  management  practices  that  are  rolled  out  with  little,  if  any,  consideration  of 
higher-order  and  unintended  consequences. 

Complex  engineered  and  natural  systems  can  be  characterized  as  complex  adaptive  systems  where 
independent,  yet  interdependent,  intelligent  agents  pursue  their  goals,  often  in  conflict  with  other 
agents,  and  learn  and  adapt  to  the  changing  ecosystem.  A  multi-level  approach  to  computationally 
modeling  the  functioning  of  such  systems  can  provide  the  means  to  understanding  and  then 
transforming  these  systems. 


Survey  of  Industry  Practices 


The  issues  raised  in  the  Terms  of  Reference  are  not  unique  to  the  Department  of  Defense.  Other,  non¬ 
defense  industries  associated  with  the  design,  development,  deployment,  operation  and  sustainment  of 
complex  systems  face  these  issues  as  well.  This  section  reports  on  a  series  of  interviews  of  eight 
executives  in  four  industries  -  automobile,  commercial  aerospace,  building  equipment,  and 
semiconductors  and  electronics.  The  results  of  these  interviews  provide  insights  that  are  likely  to  be  of 
value  as  the  DMMF  initiative  proceeds 
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Methodology 


The  process  began  with  email  contact  of  senior  executives  in  the  four  industries.  These  executives  held 
positions  ranging  from  Chief  Technology  Officer,  to  Vice  President  for  R&D,  to  Chief  Scientist.  Typically, 
these  emails  led  to  electronic  introductions  to  the  people  appropriate  for  interviews.  The  body  of  the 
email  was  as  follows: 

"I  am  a  member  of  an  advisory  panel  to  the  Assistant  Secretary  of  Defense  for  Research  and 
Engineering.  Our  task  is  to  assess  the  state  of  the  art  in  multi-level  modeling  in  general  and  model 
composition  in  particular.  The  overarching  question  concerns  the  prospects  of  connecting  computational 
models  developed  at  different  times  for  different  purposes  to  answer  new  guestions  such  as  the  impacts 
of  new  technological  capabilities  or  perhaps  new  operational  policies. 

One  of  my  assignments  is  to  assess  how  this  need  is  met  in  non-defense  industries  such  as  commercial 
aircraft,  automobiles  and  semiconductors.  I  would  like  your  help  to  find  people  in  your  industry  that  can 
help  me  understand  how  they  go  about  combining  legacy  models  into  composite  models  for  addressing 
new  questions.  I  expect  that  this  is  easiest  for  models  of  physical  systems  such  as  an  engine  or  circuit 
board,  and  gets  more  difficult  when  looking  at  whole  vehicles,  or  manufacturing  of  systems  or 
subsystems.  My  sense  is  that  this  is  most  challenging  when  behavioral  and  social  phenomena  are  central 
to  system  performance. 

This  endeavor  also  raises  questions  of  "curation"  of  computational  models,  including  assumption 
management  when  creating  composite  models.  The  value  of  legacy  models  would  seem  to  depend 
heavily  on  how  these  issues  are  managed.  I  am  also  interested  in  how  your  industry  handles  such  issues. 

If  you  do  not  know  of  someone  in  your  industry  who  works  in  the  area  of  combining  legacy  models,  it 
would  still  be  helpful  for  me  to  speak  with  someone  who  is  a  thought  leader  in  modeling  complex 
systems  to  gain  an  understanding  of  important  issues  and  unmet  needs. 

Thank  you  in  advance  for  helping  me  to  find  some  people  to  talk  with  about  these  issues." 

Subsequent  telephone  interviews  were  scheduled.  Each  interview  was  roughly  30  minutes  in  length. 
The  central  questions  asked  were  as  follows: 

To  what  extent  is  composition  of  legacy  models  an  issue  for  you? 

What  types  of  models  are  composed? 

What  socio-technical  phenomena  are  included?  Operations?  Maintenance? 
Manufacturing? 

Who  else  do  you  recommend  we  contact? 

All  of  the  interviews  evolved  into  open-ended  discussions.  Interview  notes  were  captured  in  written 
form  and  then  edited  into  electronic  form. 
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Results  of  Interviews 


Table  2  provides  a  summary  of  the  interview  results.  All  of  these  four  industries  use  modeling  and 
simulation  at  multiple  levels  of  abstraction.  These  is,  however,  little  computational  linkages  of  these 
representations.  This  is  due  in  part  to  the  allocation  of  responsibilities  and  resources  across 
organizational  functions.  There  is  recognition  that  a  solution  composed  of  a  set  of  local  optima  may  not 
represent  the  globally  optimal  solution,  but  these  large  companies  have  difficulty  approaching  their 
design  and  development  activities  in  other  than  a  reductionist  manner. 

All  of  these  companies  consider  socio-technical  phenomena  to  be  important  to  the  success  of  their 
systems.  Thus,  they  address  human  behavior  and  performance,  social  and  organizational  interactions 
and  economic  decisions  making,  e.g.,  for  airline  managers  and  building  managers.  However,  these 
models  are  seldom  computationally  integrated  with  the  physics-based  models  of  the  technology 
components  of  their  systems. 

These  companies  invest  in  modeling  and  simulation  capabilities  because  they  intend  to  manufacture 
hundreds,  thousands,  or  millions  of  the  systems  they  design.  Further,  they  are  responsible  for  the 
consequences  of  design  inadequacies  or  failures.  Modeling  and  simulation  helps  them  to  make  better 
choices,  and  understand  the  consequences  of  these  choices. 

For  the  most  part,  they  all  employ  commercially  available  software  (i.e.,  Matlab,  Simulink,  Model 
Center)  as  their  computational  engines,  with  their  system  representations  being  their  proprietary 
information.  They  approach  reuse  of  models  with  caution,  in  particular  being  wary  of  "model-induced 
design  flaws."  They  impose  standards  on  tools,  software  and  documentation.  Nevertheless,  they  report 
that  cross-functional  negotiations  on  models  and  simulations  are  quite  common. 


Observations 


The  overall  DMMF  study  focused  on  the  possibility  of  addressing  new  issues  associated  with  complex 
systems  by  composing  computational  models  originally  developed  for  different  purposes  into  an  overall 
computational  tool.  This  is  a  very  difficult  problem  that  involves  many  issues  beyond  whether  two 
disparate  pieces  of  software  can  be  plugged  together  and  run  appropriately. 

A  particularly  difficult  aspect  of  such  an  endeavor  concerns  assuring  that  the  sets  of  assumptions 
associated  with  the  elements  of  the  composition  are  compatible.  These  concerns  range  from  units  of 
measure  to  coordinate  systems  to  assumptions  regarding  environmental  factors,  independence  of 
computations,  etc.  All  of  these  considerations  affect  the  validity  of  any  composition. 
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Industry 

Companies 

Levels  of  Modeling  & 
Simulation 

Human 

Phenomena 

How  Models 
Are  Composed 

Re-Use 

Standards 

Automobile 

General 

Automobile  or 

Drivers 

Focused  reuse 

Design  for 

Motors 

drivetrain 

Passengers 

Proprietary 

reuse 

Toyota 

Performance 

Operations 

(driver) 

Traffic  (intra 
vehicle) 

Assemblers 

Maintainers 

representations 

Commercial 

computations 

Compatible 

abstractions 

& 

assumptions 

Commercial 

Boeing 

Aircraft  or 

Pilots 

Product  line 

Standards  for 

Aerospace 

Pratt  & 

engine 

Passengers 

reuse 

work, 

Whitney 

Performa 

nee 

Operation 
s  (e.g.,  range) 
Market 
(e.g.,  routes) 

Assemblers 

Maintainers 

Owners 

Proprietary 

representations 

Commercial 

computations 

software 

Building 

Carrier 

El 

Residents 

Focused  reuse 

Standards  for 

Equipment 

Otis 

evator  or  AC 

C 

ontrol  system 

H 

uman  demand 

B 

uilding  evacuation 
En 

ergy  consumption 

Customers 

Managers 

Proprietary 

representations 

Commercial 

computations 

design  tools, 
software  & 

documentatio 

n 

Semiconductors 

Freescale 

Semiconductor 

Assemblers 

Product  line 

Embodied  in 

&  Electronics 

Multek 

Circuit 

Device 

System  (e.g.,  cell 
phone) 

reuse 

Proprietary 

representations 

Commercial 

computations 

vendor- 
provided 
software  tools 

Table  2.  Multi-Level  Modeling  Across  Industries 


The  status  quo  in  DoD  is  that  such  composition  is  accomplished,  but  very  slowly  and  very  expensively. 
The  real  time  composition  capabilities  requested  in  the  Terms  of  Reference  are  rarely  available.  Non¬ 
defense  industries  do  have  some  of  these  capabilities.  However,  investments  in  such  capabilities  have 
been  limited  to  situations  where  these  investments  can  be  amortized  across  large  numbers  of  units 
produced. 
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To  the  extent  that  "plug  and  play"  of  models  is  possible  —  and  meaningful  --  the  component  models 
have  usually  been  designed  to  be  composed,  in  contrast  to  being  re-purposed  from  pieces  not  originally 
meant  for  composition.  Design  for  re-use  often  involves  adopting  standards  for  software  development 
and  documentation. 

It  should  also  be  noted  that  the  time  and  money  required  for  DoD  to  compose  models  for  new  purposes 
are  not  solely  due  to  the  difficulty  of  the  technical  problem.  DoD  business  processes  for  securing 
resources  and  requesting  permissions  have  been  found,  at  least  in  some  cases,  to  consume  the  majority 
of  the  time  and  money. 

Conclusions 


The  problem  of  composing  disparate  computational  models  into  composite  tools  for  addressing  new 
problems  is  pervasive  across  DoD  and  non-defense  industries.  It  is  very  difficult  to  assure  the  validity  of 
the  resulting  combination.  The  bottom  line  is  that  "plug  and  play"  is  immensely  difficult  without 
standards  for  code,  documentation,  etc.  When  the  consequences  are  sufficient,  however,  this  capability 
is  feasible. 


Contract  Number:  H98230-08-D-01 71  TO  0023,  RT  041 

Report  No.  SERC-2012-TR-020 
Octobers,  2012 

UNCLASSIFIED 

13 


